{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Action space, Policy, Episode, Horizon\n",
    "\n",
    "In this section, we will learn about the several important fundamental concepts that are involved in reinforcement learning. \n",
    "\n",
    "## Action space\n",
    "Consider the grid world environment shown below:\n",
    "\n",
    "![title](Images/16.png)\n",
    "\n",
    "\n",
    "In the preceding grid world environment, the goal of the agent is to reach state I\n",
    "starting from state A without visiting the shaded states. In each of the states, the\n",
    "agent can perform any of the four actions—up, down, left, and right—to achieve the\n",
    "goal. The set of all possible actions in the environment is called the action space.\n",
    "Thus, for this grid world environment, the action space will be [up, down, left, right].\n",
    "We can categorize action spaces into two types:\n",
    "\n",
    "* Discrete action space \n",
    "* Continuous action space\n",
    "\n",
    "__Discrete action space__ -When our action space consists of actions that are discrete,\n",
    "then it is called a discrete action space. For instance, in the grid world environment,\n",
    "our action space consists of four discrete actions, which are up, down, left, right, and\n",
    "so it is called a discrete action space.\n",
    "\n",
    "__Continuous action space__ - When our action space consists of actions that are\n",
    "continuous, then it is called a continuous action space. For instance, let's suppose\n",
    "we are training an agent to drive a car, then our action space will consist of several\n",
    "actions that have continuous values, such as the speed at which we need to drive the\n",
    "car, the number of degrees we need to rotate the wheel, and so on. In cases where\n",
    "our action space consists of actions that are continuous, it is called a continuous\n",
    "action space.\n",
    "\n",
    "## Policy\n",
    "\n",
    "A policy defines the agent's behavior in an environment. The policy tells the agent\n",
    "what action to perform in each state. For instance, in the grid world environment, we\n",
    "have states A to I and four possible actions. The policy may tell the agent to move\n",
    "down in state A, move right in state D, and so on.\n",
    "\n",
    "To interact with the environment for the first time, we initialize a random policy, that\n",
    "is, the random policy tells the agent to perform a random action in each state. Thus,\n",
    "in an initial iteration, the agent performs a random action in each state and tries to\n",
    "learn whether the action is good or bad based on the reward it obtains. Over a series\n",
    "of iterations, an agent will learn to perform good actions in each state, which gives a\n",
    "positive reward. Thus, we can say that over a series of iterations, the agent will learn\n",
    "a good policy that gives a positive reward.\n",
    "\n",
    "The optimal policy is shown in the following figure. As we can observe, the agent selects the\n",
    "action in each state based on the optimal policy and reaches the terminal state I from\n",
    "the starting state A without visiting the shaded states:\n",
    "\n",
    "![title](Images/17.png)\n",
    "\n",
    "Thus, the optimal policy tells the agent to perform the correct action in each state so\n",
    "that the agent can receive a good reward.\n",
    "\n",
    "A policy can be classified into two:\n",
    "\n",
    "* Deterministic Policy\n",
    "* Stochastic Policy\n",
    "\n",
    "### Deterministic Policy\n",
    "The policy which we just learned above is called deterministic policy. That is, deterministic policy tells the agent to perform a one particular action in a state. Thus, the deterministic policy maps the state to one particular action and is often denoted by $\\mu$. Given a state $s$ at a time $t$, a deterministic policy tells the agent to perform a one particular action $a$. It can be expressed as:\n",
    "\n",
    "$$a_t = \\mu(s_t) $$\n",
    "\n",
    "For instance, consider our grid world example, given a state A, the deterministic policy $\\mu$ tells the agent to perform an action down and it can be expressed as:\n",
    "\n",
    "$$\\mu (A) = \\text{Down} $$\n",
    "\n",
    "Thus, according to the deterministic policy, whenever the agent visits state A, it performs the action down. \n",
    "\n",
    "### Stochastic Policy\n",
    "\n",
    "Unlike deterministic policy, the stochastic policy does not map the state directly to one particular action, instead, it maps the state to a probability distribution over an action space. \n",
    "\n",
    "That is, we learned that given a state, the deterministic policy will tell the agent to perform one particular action in the given state, so, whenever the agent visits the state it always performs the same particular action. But with stochastic policy, given a state, the stochastic policy will return a probability distribution over an action space so instead of performing the same action every time the agent visits the state, the agent performs different actions each time based on a probability distribution returned by the stochastic policy. \n",
    "\n",
    "Let's understand this with an example, we know that our grid world environment's action space consists of 4 actions which are [up, down, left, right]. Given a state A, the stochastic policy returns the probability distribution over the action space as [0.10,0.70,0.10,0.10]. Now, whenever the agent visits the state A, instead of selecting the same particular action every time, the agent selects the action up 10% of the time, action down 70% of the time, action left 10% of time and action right 10% of the time. \n",
    "\n",
    "The difference between the deterministic policy and stochastic policy is shown below, as we can observe the deterministic policy maps the state to one particular action whereas the stochastic policy maps the state to the probability distribution over an action space:\n",
    "\n",
    "![title](Images/18.png)\n",
    "\n",
    "Thus, stochastic policy maps the state to a probability distribution over action space and it is often denoted by $\\pi$.  Say, we have a state $s$ and action $a$ at a time $t$, then we can express the stochastic policy as:\n",
    "\n",
    "\n",
    "$$a_t \\sim \\pi(s_t) $$\n",
    "\n",
    "Or it can also be expressed as $\\pi(a_t |s_t) $. \n",
    "\n",
    "We can categorize the stochastic policy into two:\n",
    "\n",
    "* Categorical policy\n",
    "* Gaussian policy\n",
    "\n",
    "### Categorical policy \n",
    "A stochastic policy is called a categorical policy when the action space is discrete. That is, the stochastic policy uses categorical probability distribution over action space to select actions when the action space is discrete. For instance, in the grid world environment, we have just seen above, we select actions based on categorical probability distribution (discrete distribution) as the action space of the environment is discrete. As shown below, given a state A, we select an action based on the categorical probability distribution over the action space:\n",
    "\n",
    "\n",
    "\n",
    "![title](Images/19.png)\n",
    "### Gaussian policy \n",
    "A stochastic policy is called a gaussian policy when our action space is continuous. That is, the stochastic policy uses Gaussian probability distribution over action space to select actions when the action space is continuous. Let's understand this with a small example. Suppose we training an agent to drive a car and say we have one continuous action in our action space. Let the action be the speed of the car and the value of the speed of the car ranges from 0 to 150 kmph. Then, the stochastic policy uses the Gaussian distribution over the action space to select action as shown below:\n",
    "\n",
    "![title](Images/20.png)\n",
    "\n",
    "\n",
    "We will learn more about the gaussian policy in the upcoming chapters.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Episode \n",
    "\n",
    "The agent interacts with the environment by performing some action starting from the initial state and reach the final state. This agent-environment interaction starting from the initial state until the final state is called an episode. For instance, in the car racing video game, the agent plays the game by starting from the initial state (starting point of the race) and reach the final state (endpoint of the race). This is considered an episode. An episode is also often called trajectory (path taken by the agent) and it is denoted by $\\tau$. \n",
    "\n",
    "An agent can play the game for any number of episodes and each episode is independent of each other. What is the use of playing the game for multiple numbers of episodes? In order to learn the optimal policy, that is, the policy which tells the agent to perform correct action in each state, the agent plays the game for many episodes. \n",
    "\n",
    "For example, let's say we are playing a car racing game, for the first time, we may not win the game and we play the game several times to understand more about the game and discover some good strategies for winning the game. Similarly, in the first episode, the agent may not win the game and it plays the game for several episodes to understand more about the game environment and good strategies to win the game. \n",
    "\n",
    "\n",
    "\n",
    "Say, we begin the game from an initial state at a time step t=0 and reach the final state at a time step T then the episode information consists of the agent environment interaction such as state, action, and reward starting from the initial state till the final state, that is, $(s_0, a_0,r_0,s_1,a_1,r_1,\\dots,s_T) $\n",
    "\n",
    "An episode (or) trajectory is shown below:\n",
    "\n",
    "![title](Images/21.png)\n",
    "\n",
    "\n",
    "Let's strengthen our understanding of the episode and optimal policy with the grid world environment. We learned that in the grid world environment, the goal of our agent is to reach the final state I starting from the initial state A without visiting the shaded states. An agent receives +1 reward when it visits the unshaded states and -1 reward when it visits the shaded states.\n",
    "\n",
    "When we say, generate an episode it means going from initial state to the final state. The agent generates the first episode using a random policy and explores the environment and over several episodes, it will learn the optimal policy. \n",
    "\n",
    "### Episode 1:\n",
    "\n",
    "As shown below, in the first episode, the agent uses random policy and selects random action in each state starting from the initial state until the final state and observe the reward:\n",
    "\n",
    "\n",
    "![title](Images/22.png)\n",
    "\n",
    "\n",
    "### Episode 2:\n",
    "\n",
    "In the second episode, the agent tries a different policy to avoid negative rewards which it had received in the previous episode. For instance, as we can observe in the previous episode, the agent selected an action right in the state A and received a negative reward, so in this episode, instead of selecting action right in the state A, it tries a different action say, down as shown below:\n",
    "\n",
    "\n",
    "![title](Images/23.png)\n",
    "\n",
    "### Episode n:\n",
    "\n",
    "Thus, over a series of the episodes, the agent learns the optimal policy, that is, the policy which takes the agent to the final state I from the state A without visiting the shaded states as shown below:\n",
    "\n",
    "\n",
    "![title](Images/24.png)\n",
    "\n",
    "# Episodic and Continuous tasks \n",
    "A reinforcement learning task can be categorized into two:\n",
    "* Episodic task\n",
    "* Continuous task\n",
    "\n",
    "__Episodic task__ - As the name suggests episodic task is the one that has the terminal state. That is, episodic tasks are basically tasks made up of episodes and thus they have a terminal state. Example: Car racing game. \n",
    "\n",
    "__Continuous task__ - Unlike episodic tasks, continuous tasks do not contain any episodes and so they don't have any terminal state. For example, a personal assistance robot does not have a terminal state. \n",
    "\n",
    "\n",
    "# Horizon\n",
    "Horizon is the time step until which the agent interacts with the environment. We can classify the horizon into two:\n",
    "\n",
    "* Finite horizon\n",
    "* Infinite horizon\n",
    "\n",
    "__Finite horizon__ - If the agent environment interaction stops at a particular time step then it is called finite Horizon. For instance, in the episodic tasks agent interacts with the environment starting from the initial state at time step  t =0 and reach the final state at a time step T.  Since the agent environment interaction stops at the time step T, it is considered a finite horizon. \n",
    "\n",
    "__Infinite horizon__ - If the agent environment interaction never stops then it is called an infinite horizon. For instance, we learned that the continuous task does not have any terminal states, so the agent environment interaction will never stop in the continuous task and so it is considered an infinite horizon. \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
